Abstract Gait recognition methods have difficulty in achieving satisfactory performance, since the gait is vulnerable to covariates such as occlusion, clothing, view angles and carrying condition. Based on the framework of end-to-end learning and multi-layer feature extraction technology, fruitful achievements are made by applying deep learning to the field of gait recognition. The status quo, pros and cons of deep learning in gait recognition are reviewed, and the key technologies and several potential research directions are discussed.
Corresponding Authors:
ZHANG Junping, Ph.D., professor. His research interests include machine learning, intelligent transportation systems, biometric authentication and image processing.
About author:: HE Yiwei, master student. His research interests include machine learning, computer vision and gait recognition.
[1] HAN J, BHANU B. Individual Recognition Using Gait Energy Image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 28(2): 316-322. [2] 钟兴志,王晨升,刘 丰,等.步态识别综述.软件, 2013, 34(4): 160-164. (ZHONG X Z, WANG C S, LIU F, et al. A Survey of Gait Recognition. Software, 2013, 34(4): 160-164.) [3] 王科俊,侯本博.步态识别综述.中国图象图形学报, 2007, 12(7): 1152-1160. (WANG K J, HOU B B. A Survey of Gait Recognition. Journal of Image and Graphics, 2007, 12(7): 1152-1160.) [4] L Z W, XING X L, WANG K J, et al. Class Energy Image Analysis for Video Sensor-Based Gait Recognition: A Review. Sensors, 2015, 15(1): 932-964. [5] 贲晛烨,徐 森,王科俊.行人步态的特征表达及识别综述.模式识别与人工智能, 2012, 25(1): 71-81. (BEN X Y, XU S, WANG K J, et al. Review on Pedestrian Gait Feature Expression and Recognition. Pattern Recognition and Artificial Intelligence, 2012, 25(1): 71-81.) [6] 陈昌由,张军平.步态识别的特征提取综述.计算机研究与发展, 2007, 44(S2): 361-365. (CHEN C Y, ZHANG J P. A Survey of Feature Extraction on Gait Recognition. Journal of Computer Research and Development, 2007, 44(S2): 361-365 ) [7] HE K M, GEORGIA G, PIOTR D, et al. Mask R-CNN//Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 2980-2988. [8] BOYKOV Y Y, JOLLY M P. Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images//Proc of the 8th IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2001: 105-112. [9] BULAT A, TZIMIROPOULOS G. Human Pose Estimation via Convo-lutional Part Heatmap Regression//Proc of the European Confe-rence on Computer Vision. Berlin, Germany: Springer, 2016: 717-732. [10] CAO Z, SIMON T, WEI S E, et al. Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields//Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1302-1310. [11] TAKEMURA N, MAKIHARA Y, MURAMATSU D, et al. On Input/Output Architectures for Convolutional Neural Network-Based Cross-View Gait Recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2018. DOI 10.1109/TCSVT.2017.2760835. [12] IWAMA H, OKUMURA M, MAHIKARA Y, et al. The OU-ISIR Gait Database Comprising the Large Population Dataset and Performance Evaluation of Gait Recognition. IEEE Transactions on Information Forensics and Security, 2012, 7(5): 1511-1521. [13] MAKIHARA Y, SUZUKI A, MURAMATSU D, et al. Joint Intensity and Spatial Metric Learning for Robust Gait Recognition//Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2017: 6786-6796. [14] YU S Q, TAN D L, TAN T N. A Framework for Evaluating the Effect of View Angle, Clothing and Carrying Condition on Gait Recognition//Proc of the IEEE International Conference on Pa-ttern Recognition. Washington. USA: IEEE, 2006: 441-444. [15] SARKAR S, PHILLIPS P J, LIU Z Y, et al. The HumanID Gait Challenge Problem: Data Sets, Performance, and Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(2): 162-177. [16] NIXON M S, CATER J N, CUNADO D, et al. Automatic Gait Recognition. Motion Analysis and Tracking, 1999, 7(2): 3-6. [17] WANG L, TAN T N, NING H Z, et al. Silhouette Analysis-Based Gait Recognition for Human Identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(12): 1505-1518. [18] WANG L, TAN T N, HU W M, et al. Automatic Gait Recognition Based on Statistical Shape Analysis. IEEE Transactions on Image Processing, 2003, 12(9): 1120-1131. [19] WANG L, NING H Z, TAN T N, et al. Fusion of Static and Dynamic Body Biometrics for Gait Recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2004, 14(2): 149-158. [20] HAYFRON-ACQUAH J B, NIXON M S, CARTER J N. Automa-tic Gait Recognition by Symmetry Analysis. Pattern Recognition Letters, 2003, 24(13): 2175-2183. [21] BOBICK A F, JOHNSON A K. Gait Recognition Using Static, Activity-Specific Parameters//Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2001: 423-430. [22] BASHIR K, XIANG T, GONG S. Gait Recognition Using Gait Entropy Image//Proc of the International Conference on Crime Detection and Prevention. London, UK: IET, 2010. DOI: 10.1049/ic.2009.0230. [23] WANG C, ZHANG J P, WANG L, et al. Human Identification Using Temporal Information Preserving Gait Template. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11): 2164-2176. [24] LIU J Y, ZHENG N N. Gait History Image: A Novel Temporal Template for Gait Recognition//Proc of the IEEE International Conference on Multimedia and Expo. Washington, USA: IEEE, 2007: 663-666. [25] GUAN Y, LI C T, HU Y J. Robust Clothing-Invariant Gait Recognition//Proc of the 8th IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing. Wa-shington, USA: IEEE, 2012: 321-324. [26] HUANG S, ELGAMMAL A, LU J W, et al. Cross-Speed Gait Recognition Using Speed-Invariant Gait Templates and Globality-Locality Preserving Projections. IEEE Transactions on Information Forensics and Security, 2015, 10(10): 2071-2083. [27] SINGH S, BISWAS K K. Biometric Gait Recognition with Carrying and Clothing Variants//Proc of the 3rd International Conference on Pattern Recognition and Machine Intelligence. Berlin, Germany: Springer, 2009: 446-451. [28] ZHANG J P, PU J, CHEN C Y. Low-Resolution Gait Recognition. IEEE Transactions on Systems, Man, and Cybernetics(Cyberne-tics), 2010, 40(4): 986-996. [29] KUSAKUNNIRAN W, WU Q, ZHANG J, et al. Recognizing Gaits Across Views through Correlated Motion Co-clustering. IEEE Transactions on Image Processing, 2014, 23(2): 696-709. [30] BASHI K, XIANG T, GONG S G. Cross View Gait Recognition Using Correlation Strength//Proc of the British Machine Vision Conference. London, UK: BMVA, 2010: 1-11. [31] HU M D, WANG Y H, ZHANG Z X, et al. View-Invariant Discriminative Projection for Multi-view Gait-Based Human Identification. IEEE Transactions on Information Forensics and Security, 2013, 8(12): 2034-2045. [32] XING X L, WANG K J, YAN T, et al. Complete Canonical Correlation Analysis with Application to Multi-view Gait Recognition. Pattern Recognition, 2016, 50: 107-117. [33] MAKIHARA Y, SAGAWA R, MUKAIGAWA Y, et al. Gait Re-cognition Using a View Transformation Model in the Frequency Domain//Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2006, III: 151-163. [34] KUSAKUNNIRAN W, WU Q, ZHANG J, et al. Support Vector Regression for Multi-view Gait Recognition Based on Local Motion Feature Selection//Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2010: 974-981. [35] KUSAKUNNIRAN W, WU Q, LI H D, et al. Multiple Views Gait Recognition Using View Transformation Model Based on Optimized Gait Energy Image//Proc of the 12th IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2010: 1058-1064. [36] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet Cla-ssification with Deep Convolutional Neural Networks//Proc of the 26th Annual Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2012: 1097-1105. [37] BENGIO Y, SIMARD P Y, FRASCONI P. Learning Long-Term Dependencies with Gradient Descent Is Difficult. IEEE Transactions on Neural Networks, 1994, 5(2): 157-166. [38] ZHANG X F, SUN S Q, LI C, et al. DeepGait: A Learning Deep Convolutional Representation for Gait Recognition//Proc of the Chinese Conference on Biometric Recognition. Berlin, Germany: Springer, 2017: 447-456. [39] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2017-11-25]. https://arxiv.org/pdf/1409.1556.pdf. [40] DENG J, DONG W, SOCHER R, et al. ImageNet: A Large-Scale Hierarchical Image Database//Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2009: 248-255. [41] SHIRAGA K, MAKIHARA Y, MURAMATSU D, et al. GEINet: View-Invariant Gait Recognition Using a Convolutional Neural Network//Proc of the International Conference on Biometrics. Wa-shington, USA: IEEE, 2016. DOI: 10.1109/ICE.2016.7550060. [42] WOLF T, BABAEE M, RIGOLL G. Multi-view Gait Recognition Using 3D Convolutional Neural Networks//Proc of the IEEE International Conference on Image Processing. Washington, USA: IEEE, 2016: 4165-4169. [43] TRAN D, BOURDEV L, FERGUS R, et al.Learning Spatiotemporal Features with 3D Convolutional Networks//Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2015: 4489-4497. [44] SUN D Q, ROTH S, BLACK M J. Secrets of Optical Flow Estimation and Their Principles//Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2010: 2432-2439. [45] THAPAR D, AGGARWAL D, AGARWAL P, et al. VGR-Net: A View Invariant Gait Recognition Network[C/OL]. [2017-10-13]. https://arxiv.org/pdf/1710.04803.pdf. [46] HADSELL R, CHOPRA S, LECUN Y. Dimensionality Reduction by Learning an Invariant Mapping//Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2006: 1735-1742. [47] SCHROFF F, KALENICHENKO D, PHILBIN J. FaceNet: A Unified Embedding for Face Recognition and Clustering//Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 815-823. [48] LIAO R J, CAO C S, GARCIA E B, et al. Pose-Based Temporal-Spatial Network(PTSN) for Gait Recognition with Carrying and Clothing Variations//Proc of the Chinese Conference on Biometric Recognition. Berlin, Germany: Springer, 2017: 474-483. [49] HOCHREITER S, SCHMIDHUBER J. Long Short-Term Memory. Neural Computation, 1997, 9(8): 1735-1780. [50] GRAVES A, MOHAMED A R, HINTON G. Speech Recognition with Deep Recurrent Neural Networks//Proc of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington, USA: IEEE, 2013: 6645-6649. [51] SUTSKEVER I, VINYALS O, LE Q V. Sequence to Sequence Learning with Neural Networks//Proc of the 28th Annual Confe-rence on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2014: 3104-3112. [52] WU Z F, HUANG Y Z, WANG L, et al. A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(2): 209-226. [53] CHEN Q, WANG Y H, LIU Z, et al. Feature Map Pooling for Cross-View Gait Recognition Based on Silhouette Sequence Images[C/OL]. [2017-11-26]. https://arxiv.org/pdf/1711.0935-8.pdf. [54] FENG Y, LI Y C, LUO J B. Learning Effective Gait Features Using LSTM//Proc of the 23rd IEEE International Conference on Pattern Recognition. Washington, USA: IEEE, 2017: 325-330. [55] PFISTER T, CHARLES J, ZISSERMAN A. Flowing ConvNets for Human Pose Estimation in Videos//Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2016: 1913-1921. [56] YU S Q, CHEN H F, REYES E B G, et al. GaitGAN: Invariant Gait Feature Extraction Using Generative Adversarial Networks//Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2017: 30-37. [57] MIRZA M, OSINDERO S. Conditional Generative Adversarial Nets [C/OL]. [2014-11-06]. https://arxiv.org/pdf/1411.1784.pdf. [58] CHEN X, DUAN Y, HOUTHOOFT R, et al. InfoGAN: Interpre-table Representation Learning by Information Maximizing Generative Adversarial Nets//LEE D D, SUGIYAMA M, LUXBURG U V, et al., eds. Advances in Neural Information Processing Systems 29. Cambridge, USA: The MIT Press, 2016: 2172-2180. [59] YOO D, KIM N, PAR S, et al. Pixel-Level Domain Transfer// Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 517-532. [60] YU S Q, CHEN H F, WANG Q, et al. Invariant Feature Extraction for Gait Recognition Using Only One Uniform Model. Neurocomputing, 2017, 239: 81-93. [61] ZHENG L, BIE Z, SUN Y F, et al. Mars: A Video Benchmark for Large-Scale Person Re-identification//Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 868-884. [62] LU J W, TAN Y P. Gait-Based Human Age Estimation. IEEE Transactions on Information Forensics and Security, 2010, 5(4): 761-770. [63] YU S Q, TAN T N, HUANG K Q, et al. A Study on Gait-Based Gender Classification. IEEE Transactions on Image Processing, 2009, 18(8): 1905-1910. [64] EBENEZER R H P I. Robust Analytics for Video-Based Gait Biometrics. Ph.D. Dissertation. Chennai, India: Anna University, 2017. [65] O′SULLIVAN J D, SAID C M, DILLON L C, et al. Gait Analysis in Patients with Parkinson′s Disease and Motor Fluctuations: Influence of Levodopa and Comparison with Other Measures of Motor Function. Movement Disorders, 1998, 13(6): 900-906. [66] MURO-DE-LA-HERRAN A, GARCIA-ZAPIRAIN B, MENDEZ-ZORRILLA A. Gait Analysis Methods: An Overview of Wearable and Non-wearable Systems, Highlighting Clinical Applications. Sensors, 2014, 14(2): 3362-3394.
2018, Vol. 31 
